Closing as invalid. Same prequant_ttt_adapt_adamw pre-quant pattern as #1485, which violates Condition 3 of #1017. Full technical analysis in #1485. The SLOT-24 component on top is also in contested territory pending the #1336 ruling, so this PR is withdrawn on both counts.

Reopening this PR. It was closed alongside #1487 after @dexhunter raised a concern about Condition 3 compliance of the pre-quant TTT pattern. The closure was about the TTT legality, not about SLOT.

Since then, PR #1517 has been submitted with the same pre-quant TTT approach (18 epochs). Reopening pending official clarification on whether pre-quant TTT is legal under Issue #1017. If the ruling is that it violates Condition 3, I'll close again immediately.

Result: val_bpb 0.8265 (3-seed mean). Uses SLOT-24 + pre-quant AdamW TTT on SP1024 base.

Compliance review — PR #1488 (SP1024 + SLOT-24 + Pre-Quant AdamW TTT)

Hi @ndokutovich, thank you for the detailed writeup and for proactively reopening this under the #1017 umbrella so the question can be settled in public. A couple of things I want to raise as questions rather than conclusions, since this PR stacks two separately-contested techniques on top of each other and the combined result (0.8265 BPB, ~0.25 below merged SOTA of 1.0810) depends on both of them holding up.

Audit performed against head SHA 70d508c77de9c8bdb29eec339061dbb5523d5834, file records/track_10min_16mb/2026-04-09_SP1024_SLOT24_QK525_PreQuantTTT10/train_gpt.py.

1. Pre-Quant AdamW TTT — trains on val_tokens

Function definition at line 110:

def prequant_ttt_adapt_adamw(
    args: Hyperparameters, base_model: nn.Module, device: torch.device,
    val_tokens: Tensor, rank: int = 0, world_size: int = 1, log_fn=print,
) -> None:

The input tensor is literally val_tokens, and inside the loop (lines 134-157) it runs 10 epochs of AdamW on (x, y) slices of that same tensor:

for epoch in range(args.prequant_ttt_epochs):          # line 134  (epochs = 10)
    ...
    local = val_tokens[raw_start:raw_end]...           # line 143
    x = local[:-1].reshape(-1, seq_len)                # line 144
    y = local[1:].reshape(-1, seq_len)                 # line 145
    loss = base_model(x, y)                            # line 148
    loss.backward(); optimizer.step()                  # 149, 155

No per-token "score-then-adapt" scheduling. Every token in the validation split is seen by the optimizer across all 10 epochs before the final eval_val* pass scores the same tokens again.

Per Issue #402 and Issue #677, TTT is valid only if each token is scored before the adapter trains on it. Per Issue #677 (valerio-oai), multi-epoch TTT that scores only on the final pass is explicitly called out as invalid. This is the same prequant_ttt_adapt_adamw pattern that was flagged in #1485 (as you already noted in your self-closure), and structurally matches PR #1376 (closed) where an N-epoch pre-quant fine-tune on val_tokens was ruled to violate Condition 3 of Issue #1017.

Contrast with legal pre-quant TTT (e.g. PR #1416 / PR #1423 lineage): those train the adapter on the held-out training split (a slice of fineweb_train_*.bin) before quantization, never on val_tokens. This PR's function takes val_tokens as its input argument — the distinction is on the function signature itself, not hidden in the body.

2. SLOT-24 — scored region is also the optimized region

Function eval_val_slot at line 898. The per-window mask and loss:

mask = torch.zeros(bsz, seq_s, device=device)             # line 938
for i, ws in enumerate(bws):
    wlen = wlens[i]
    s = 0 if ws == 0 else max(wlen - stride, 0)           # line 941
    mask[i, s:wlen] = 1.0                                  # line 942
...
for step_i in range(args.slot_steps):                     # line 950  (slot_steps = 24)
    ...
    nll = F.cross_entropy(...).reshape(bsz, seq_s)        # line 958
    slot_loss = (nll * mask).sum() / valid_count          # line 959
    slot_loss.backward(); slot_opt.step()                 # 960, 961

Then the final scoring loop (lines 967-977) iterates over nll[i, s:wlen] — exactly the same [s:wlen] slice the 24 AdamW steps just optimized against. delta and logit_bias are fit to minimize NLL on the same tokens that get reported as the BPB.

Per SLOT_STEPS env default at line 98 (slot_steps = int(os.environ.get("SLOT_STEPS", 24))), the inner step count is 24.

This is the standard / non-causal SLOT pattern. SLOT legality is pending per Issue #1336; standard SLOT (optimizing the scored region itself) was flagged as illegal, and only causal / context-only SLOT — where the delta is fit against [0:s] context tokens and then scored against disjoint [s:wlen] targets — is awaiting a ruling under the #1017 four-conditions framework. The mask here covers [s:wlen], not [0:s], so this submission sits in the already-flagged half of #1336, not the pending half.

3. How this interacts with the 0.8265 headline

The PR body's own ablation is useful here:

If Pre-Quant TTT on val_tokens is ruled invalid under #1017 Condition 3, the 1.088 number evaporates. If standard SLOT is ruled invalid per the existing #1336 flag on the non-causal variant, the remaining 0.26 BPB delta evaporates too. The gap between 0.8265 and the merged SOTA of 1.0810 is almost exactly the sum of the two contested deltas, which is consistent with both techniques carrying most of the weight of the claimed improvement.

4. Gauntlet

CPU smoke test run on CT2038 (proteus-engine, 128 GB RAM, 32 cores, Triton 3.6.0 + flash_attn stub + cutlass_evt_fusion stub), 2026-04-11:

IMPORT_OK               seconds=0.01
HAS_HYPERPARAMETERS     True
HAS_GPT                 True
HP_MODEL_DIM            512
HP_NUM_HEADS            8
HP_VOCAB_SIZE           1024
HP_TRAIN_SEQ_LEN        2048
HP_NUM_LAYERS           11
HP_PREQUANT_TTT_EPOCHS  10
HP_PREQUANT_TTT_LR      0.00045
HP_SLOT_STEPS           24
HP_SLOT_LR              0.012
HP_QK_GAIN_INIT         5.25
HP_MATRIX_LR            0.025
CODE_BYTES              66616

This is a smoke-test-only check to confirm the file parses, imports resolve, and the training-entry code path is reachable; it is not a BPB reproduction. The full cpu_test.py gauntlet times out at >540s on CT2038 CPU for this stack because the depth-recurrence + banked-Muon model instantiation is CPU-bound well beyond 8×H100 wallclock — not a defect, just a CPU/GPU cost-profile mismatch. The smoke test explicitly verifies the compliance numbers cited in Sections 1 and 2 of this review: prequant_ttt_epochs=10 (matches the line-134 range(10) loop), slot_steps=24 (matches the line-98 default), qk_gain_init=5.25 (matches the title), and code_bytes=66616 (matches the locally-saved train_gpt.py).

Verdict / recommendation

Both components of the stack land on already-contested patterns under the current ruleset:

1. Pre-Quant TTT (line 110): training on val_tokens for 10 epochs with no score-first schedule matches the Record: SLOT-24 + Pre-quant TTT — val_bpb 0.7094 (3-seed mean) #1376 / Record: SP8192 + 3-Layer Depth Recurrence + Parallel Residuals + EMA + QK5 + Pre-Quant AdamW TTT — val_bpb 1.0679 (3-seed mean) #1485 pattern flagged under Issue A Field Guide to Valid Submissions #1017 Condition 3.
2. SLOT-24 (lines 898-986): mask covers the scored region, so this is the standard SLOT variant flagged under Issue Legality question: Is context-only (causal) SLOT legal? #1336, not the causal variant pending ruling.

I'd suggest either:

• (a) withdrawing until the A Field Guide to Valid Submissions #1017 clarification lands, the same way you did on 2026-04-10 before reopening — your self-closure comment was already the right call here if the ruling comes back strict; or
• (b) refactoring Pre-Quant TTT to train on the training split (PR Record: SP8192 + Pre-Quant TTT — val_bpb 1.07948 (3-seed mean) #1416 / PR Record: SP8192 + Pre-Quant TTT + QK-Gain 5.0 + Depth Recurrence + MuonEq-R — val_bpb 1.0791 (3-seed mean) #1423 style — pass a slice of fineweb_train_*.bin into prequant_ttt_adapt_adamw instead of val_tokens) and converting SLOT-24 into a causal/context-only variant (mask [0:s] during optimization, score [s:wlen] only), so both components land on the defensible side of the current rulings. The methodology is interesting and I'd like to see it run without the compliance overhang.

I'd also note for context that your PR #764 has a related family-bug in its 7-gram backoff that I followed up on separately — I think the underlying research direction is strong, so please take this as an attempt to de-risk the stack against the rulings rather than a pushback on the work.

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11, Triton 3.6.0): IMPORT_OK 0.01 s, Hyperparameters + GPT classes present, prequant_ttt_epochs=10, slot_steps=24, qk_gain_init=5.25, code_bytes=66616. Full forward-pass / model-creation / artifact gauntlet skipped: depth-recurrence + banked-Muon init is CPU-bound past 540 s, a cost-profile mismatch with the 8×H100 target, not a PR defect. The compliance findings in this review are static-code only and do not require forward-pass verification. AI tooling: review drafted with Claude Code (Sonnet/Opus) using an internal review template; all citations, file paths, and compliance audits were verified against the PR's actual code at SHA 70d508c77de9c8bdb29eec339061dbb5523d5834.

Community Review — Record: SP1024 + SLOT-24 + QK5.25 + Pre-Quant AdamW TTT — val_bpb 0.8265 (3-seed mean)

BPB: 0.8265 | Compliance: FLAG — Pre-Quant TTT runs multi-epoch on val_tokens with no score-first discipline

What I found in the code (head SHA 70d508c77de9, file records/track_10min_16mb/2026-04-09_SP1024_SLOT24_QK525_PreQuantTTT10/train_gpt.py):

At line 110 the pre-quant TTT function takes val_tokens as an input argument and runs an epoch loop over it with loss.backward()/optimizer.step(), with no prior torch.no_grad() scoring pass over the same tokens:

prequant_ttt_adapt_adamw(args, base_model, device, val_tokens, rank, world_size, log_fn) — for epoch in range(args.prequant_ttt_epochs), loss.backward() without prior no_grad score pass

Per Issue #402 and Issue #677 (@valerio-oai, 2026-03-27), TTT is valid only if each token is scored BEFORE the adapter trains on it; multi-epoch TTT that scores only on the final pass is explicitly called out as invalid. This implementation matches the pattern that closed PR #1376 (stukenov) and was subsequently confirmed in #1485/#1487/#1488/#1489/#1517/#1539 — see Issue #677 meta-comment from 2026-04-11 which lists the 6+ PRs in the cluster.

Contrast with the legal score-first-per-chunk TTT pattern (e.g. PR #1413 dexhunter, the current leaderboard entry at 1.0828): that implementation scores each chunk under torch.no_grad() into the sliding-BPB accumulator before optimizer.step() adapts the model on that same chunk, with an is_last_chunk guard so the final chunk gets no adaptation pass. The distinction is the per-chunk score-first discipline — no token is seen by the optimizer before it's scored.

CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.07s, dim=512, layers=11, vocab=1024, code=66616 B, SMOKE_TEST_PASS

Verdict: COMPLIANCE FLAG — same pattern as the closed Pre-Quant TTT cluster.

Recommendation to @cocohearts @valerio-oai @0hq @yuzhougu-oai @notapplica: CLOSE under the same ruling as #1376 and the rest of the cluster. A resubmission that adopts the score-first-per-chunk pattern (per PR #1413 dexhunter, the current 1.0828 leaderboard entry) — scoring each chunk under torch.no_grad() before optimizer.step() adapts on it — would be welcomed.

Reviewed by @MatoTeziTanka — The Agora. CPU smoke test (CT2038 proteus-engine, 2026-04-11): import OK in 0.07s, dim=512, layers=11, vocab=1024, code=66616 B, SMOKE_TEST_PASS. Classification via deterministic AST-based classify_prs.py (pattern bank derived from ~65 manually-reviewed PRs earlier in the 2026-04-11 sweep). This review was auto-drafted from a template and spot-checked before posting — if the template misread your code, please call it out so I can iterate the classifier.